AWS S3兼容文档

前言

介绍

本文档主要是针对NOS兼容AWS S3协议后，与S3的一些异同

支持的SDK

• Java SDK
• GO SDK
• Python SDK
• PHP SDK

支持的接口

Bucket

• List Buckets
• Put Bucket
• Put Bucket Acl
• Get Bucket Acl
• Delete Bucket
• Get Bucket Location

Object

• Put Object
• Put Object Copy
• Get Object
• Delete Object
• Delete Multiple Objects
• Head Object
• List Objects

Multipart Upload

• Initiate Multipart Upload
• Upload Part
• List Parts
• List Multipart Uploads
• Complete Multipart Upload
• Abort Multipart Upload

约束说明

参数	详细说明
加密	暂不支持
Acl	支持public-read,private
Grant	不支持
Lifecycle	不支持
多版本	不支持
Region	支持us-east-1对应的分区为杭州
StorageClass	默认为STANDARD,其他的存储级别暂不支持
Authorization	支持v4,v2，目前只支持在header加入签名信息，暂不支持url签名
MFA授权	不支持

Bucket接口

List Buckets

兼容说明

接口和S3兼容

代码示例

Java:

AWSCredentials credentials= new BasicAWSCredentials("ak","sk");
AmazonS3 s3Client = new AmazonS3Client(credentials);
s3Client.setEndpoint("http://xxx.com");//指定为nos的endpoint

List<Bucket> listBuckets = s3Client.listBuckets();
for(Bucket bucket : listBuckets){
    System.out.println(bucket.getOwner() + "-->" + bucket.getName() + "-->" +  bucket.getCreationDate());
}

Put Bucket

兼容说明

注意约束说明中的说明即可

代码示例

Java:

s3Client.createBucket("testbucket");

Put Bucket Acl

兼容说明

注意约束说明中的说明即可

代码示例

Java:

s3Client.setBucketAcl(bucketName,CannedAccessControlList.PublicRead);

Get Bucket Acl

兼容说明

• 因为NOS的权限不能赋给其他的用户，所以返回的Body中只有一条记录
• 在返回值中，Permission的取值有两种类型，Read(public-read)，Write(private)

代码示例

Java:

AccessControlList accessControlList = s3Client.getBucketAcl(bucketName);
System.out.println("owner : " +  accessControlList.getOwner().getId() + " : " + accessControlList.getOwner().getDisplayName());
for(Grant grant : accessControlList.getGrantsAsList()){//由于不支持授权给其他用户，所以List中只存在一个Grant
    System.out.println(grant.getGrantee().getIdentifier() + " : " +  grant.getPermission() + " : " + grant.getGrantee().getTypeIdentifier());
}

Delete Bucket

兼容说明

接口和S3兼容

代码示例

Java:

s3Client.deleteBucket(testBucketName);

Get Bucket Location

兼容说明

接口和S3兼容

代码示例

Java:

System.out.println(s3Client.getBucketLocation(testBucketName));

Object接口

Put Object

兼容说明

• x-amz-tagging 不支持
• x-amz-website-redirect-location 不支持
• Expires 不支持
• Content-Encoding 不支持
• Java SDK不支持http上传

代码示例

Java:

String content = "test put object content";
String testKey = "testKey";
PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName,testKey,"/");
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(content.length());
Map<String,String> userMeta = new TreeMap<>();
userMeta.put("userMeta1","meta1");
userMeta.put("userMeta2","meta2");
objectMetadata.setUserMetadata(userMeta);
putObjectRequest.setMetadata(objectMetadata);
putObjectRequest.setInputStream(new ByteInputStream(content.getBytes(Charset.forName("UTF-8")),content.getBytes(Charset.forName("UTF-8")).length));
s3Client.putObject(putObjectRequest);

Put Object Copy

兼容说明

• x-amz-metadata-directive,不支持设置，用户原信息会复制过去
• x-amz-copy-xxx 等参数不支持
• x-amz-storage-class，存储级别使用默认(STANDARD)的就好，NOS不支持其他的存储级别
• x-amz-tagging-directive,x-amz-website-redirect-location 也不支持

代码示例

Java:

CopyObjectResult copyObjectResult = s3Client.copyObject(srcBucket,srcKey,destBucket,destKey);
System.out.println(copyObjectResult.getETag());
System.out.println(copyObjectResult.getLastModifiedDate());

Get Object

兼容说明

• NOS不支持的操作，相关的响应头是不会返回的
• If-Unmodified-Since，If-Match,If-None-Match 不支持

代码示例

Java:

String testKey = "testKey";
GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName,testKey);
S3Object s3Object = s3Client.getObject(getObjectRequest);
System.out.println(s3Object.getObjectMetadata().getContentLength());
if(s3Object != null){
   S3ObjectInputStream s3ObjectInputStream = s3Object.getObjectContent();
   byte[] buf = new byte[1024];
   int readLen;
   try {
       while((readLen = s3ObjectInputStream.read(buf)) > -1){
           System.out.println(new String(buf,0,readLen));
       }
       System.out.println(s3Object.getObjectMetadata().getUserMetadata());
   } catch (IOException e) {
       e.printStackTrace();
   }
}

Delete Object

兼容说明

• NOS会直接删除对象
• 删除对象只需要正确的ak/sk
• 返回值中不会包含x-amz-delete-marker和x-amz-version-id

代码示例

Java:

s3Client.deleteObject(bucketName,testKey);

Delete Multiple Objects

兼容说明

接口和S3兼容

代码示例

Java:

String testBucketName  = bucketName;
String testKey = "test";
for(int i = 0;i < 5;++i){
    s3Client.putObject(testBucketName,testKey + i,"test" + i);
}
DeleteObjectsRequest deleteObjectsRequest = new DeleteObjectsRequest(testBucketName);
List<DeleteObjectsRequest.KeyVersion> list = new LinkedList<>();
for(int i = 0; i < 3;++i){
    list.add(new DeleteObjectsRequest.KeyVersion(testKey + i));
}
deleteObjectsRequest.setKeys(list);
deleteObjectsRequest.setQuiet(true);
DeleteObjectsResult deleteObjectsResult = s3Client.deleteObjects(deleteObjectsRequest);
for(DeleteObjectsResult.DeletedObject deleteObject : deleteObjectsResult.getDeletedObjects()){
    System.out.println(deleteObject.getKey());
}

Head Object

兼容说明

• 和GetObject一样，也不支持If-Unmodified-Since，If-Match,If-None-Match参数
• 同样的，加密也不支持
• 返回中带有Object该有的信息

代码示例

Java:

ObjectMetadata objectMetadata =  s3Client.getObjectMetadata(bucketName,testKey);
System.out.println(objectMetadata.getUserMetadata());
System.out.println(objectMetadata.getContentLength());
System.out.println(objectMetadata.getETag());

List Objects

兼容说明

S3的List Objects有两个版本(v1,v2),NOS目前只兼容v1

代码示例

Java:

ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
listObjectsRequest.setBucketName("sdktest-private");
List<S3ObjectSummary> listResult = new ArrayList<>();
ObjectListing listObjects = s3Client.listObjects(listObjectsRequest);
do {
    listResult.addAll(listObjects.getObjectSummaries());
    if (listObjects.isTruncated()) {
        ListObjectsRequest request = new ListObjectsRequest();
        request.setBucketName(listObjectsRequest.getBucketName());
        request.setMarker(listObjects.getNextMarker());
        listObjects =  s3Client.listObjects(request);
    } else {
        break;
    }
} while (listObjects != null);

Multipart Upload接口

Initiate Multipart Upload

兼容说明

x-amz-website-redirect-location，不支持

代码示例

Java:

InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(
               bucketName, key);
InitiateMultipartUploadResult initResponse =
               s3Client.initiateMultipartUpload(initRequest);

Upload Part

兼容说明

接口和S3兼容

代码示例

Java:

UploadPartRequest uploadRequest = new UploadPartRequest()
                       .withBucketName(bucketName).withKey(key)
                       .withUploadId(initResponse.getUploadId()).withPartNumber(i)
                       .withFileOffset(filePosition)
                       .withFile(file)
                       .withPartSize(partSize);

s3Client.uploadPart(uploadRequest)

List Parts

兼容说明

• 对于s3中的encoding头，如果包含了，也不处理
• 因为nos没有生命周期的概念，所以响应中不会加入lifecycle相关的响应头

代码示例

Java:

ListPartsRequest listPartsRequest = new ListPartsRequest(bucketName,key,initResponse.getUploadId());
//initResponse通过上面的init方式获取
s3Client.listParts(listPartsRequest);

List Multipart Uploads

兼容说明

• request只支持max-uploads,key-marker参数
• response中，因为不支持有的请求参数，所以有的元素也不会返回,详细见下表：

resp header	S3	NOS
UploadIdMarker	√	x
NextUploadIdMarker	√	x
Encoding-Type	√	x
Initiator	√	x
CommonPrefixes	√	x
CommonPrefixes.Prefix	√	x

代码示例

Java:

ListMultipartUploadsRequest listMultipartUploadsRequest = new ListMultipartUploadsRequest(bucketName);
listMultipartUploadsRequest.setMaxUploads(2);
MultipartUploadListing multipartUploadListing  = s3Client.listMultipartUploads(listMultipartUploadsRequest);
System.out.println("bucketName : " + multipartUploadListing.getBucketName());
System.out.println(multipartUploadListing.getMaxUploads());
for(MultipartUpload multipartUpload : multipartUploadListing.getMultipartUploads()){
     System.out.println(multipartUpload.getInitiated() + " : " + multipartUpload.getUploadId() + " : " +
                   multipartUpload.getStorageClass());
}

Complete Multipart Upload

兼容说明

除了版本控制和加密，NOS完全兼容S3

代码示例

Java:

CompleteMultipartUploadRequest compRequest = new
       CompleteMultipartUploadRequest(bucketName,
       key,
       initResponse.getUploadId(),//为initiateMultipartUpload的返回值
       partETags); //partEtags通过上述ListParts构造

Abort Multipart Upload

兼容说明

接口和S3兼容

代码示例

Java:

s3Client.abortMultipartUpload(new AbortMultipartUploadRequest(
                   bucketName, key, initResponse.getUploadId()));//initResponse为initiateMultipartUpload的返回值

完整的分块上传示例

Java:

List<PartETag> partETags = new ArrayList<>();
// Step 1: Initialize.
InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(
       bucketName, key);
InitiateMultipartUploadResult initResponse =
       s3Client.initiateMultipartUpload(initRequest);
File file = new File(filePath);
long contentLength = file.length();
long partSize = 5 * 1024 * 1024; // Set part size to 5 MB.
try {
   // Step 2: Upload parts.
   long filePosition = 0;
   for (int i = 1; filePosition < contentLength; i++) {
       // Last part can be less than 5 MB. Adjust part size.
       partSize = Math.min(partSize, (contentLength - filePosition));

       // Create request to upload a part.
       UploadPartRequest uploadRequest = new UploadPartRequest()
               .withBucketName(bucketName).withKey(key)
               .withUploadId(initResponse.getUploadId()).withPartNumber(i)
               .withFileOffset(filePosition)
               .withFile(file)
               .withPartSize(partSize);
       // Upload part and add response to our list.
       partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
       ListPartsRequest listPartsRequest = new ListPartsRequest(bucketName,key,initResponse.getUploadId());
       s3Client.listParts(listPartsRequest);

       filePosition += partSize;
   }
   // Step 3: Complete.
   CompleteMultipartUploadRequest compRequest = new
           CompleteMultipartUploadRequest(bucketName,
           key,
           initResponse.getUploadId(),
           partETags);
   s3Client.completeMultipartUpload(compRequest);
} catch (Exception e) {
   //error,abort
   s3Client.abortMultipartUpload(new AbortMultipartUploadRequest(
           bucketName, key, initResponse.getUploadId()));
}